Evaluation for Darpa Communicator Spoken Dialogue Systems
نویسندگان
چکیده
The overall objective of the DARPA COMMUNICATOR project is to support rapid, cost-effective development of multi-modal speechenabled dialogue systems with advanced conversational capabilities, such as plan optimization, explanation and negotiation. In order to make this a reality, we need to find methods for evaluating the contribution of various techniques to the users’ willingness and ability to use the system. This paper reports on the approach to spoken dialogue system evaluation that we are applying in the COMMUNICATOR program. We describe our overall approach, the experimental design, the logfile standard, and the metrics applied in the experimental evaluation planned for June of 2000.
منابع مشابه
Quantitative and Qualitative Evaluation of Darpa Communicator Spoken Dialogue Systems
This paper describes the application of the PARADISE evaluation framework to the corpus of 662 human-computer dialogues collected in the June 2000 Darpa Communicator data collection. We describe results based on the standard logfile metrics as well as results based on additional qualitative metrics derived using the DATE dialogue act tagging scheme. We show that performance models derived via u...
متن کاملOverview on Recent Activities in Speech Systems Evalua
This paper discusses issues in evaluating spoken language dialogue systems in terms of technical performance and end-user acceptance. Recent efforts in this domain have been carried out in the framework of two major research initiatives: the European Esprit long-term project Spoken Language Dialogue Systems and Components Best practice in development and evaluation (DISC) and the US American DA...
متن کاملWhat’s the Trouble: Automatically Identifying Problematic Dialogues in DARPA Communicator Dialogue Systems
Spoken dialogue systems promise efficient and natural access to information services from any phone. Recently, spoken dialogue systems for widely used applications such as email, travel information, and customer care have moved from research labs into commercial use. These applications can receive millions of calls a month. This huge amount of spoken dialogue data has led to a need for fully au...
متن کاملEffects of Speech Recognition Accuracy on the Performance of DARPA Communicator Spoken Dialogue Systems
The DARPA Communicator program explored ways to construct better spoken-dialogue systems, with which users interact via speech alone to perform relatively complex tasks such as travel planning. During 2000 and 2001 two large data sets were collected from sessions in which paid users did travel planning using the Communicator systems that had been built by eight research groups. The research gro...
متن کاملWhat's the Problem: Automatically Identifying Problematic Dialogues in DARPA Communicator Dialogue Systems
Spoken dialogue systems promise efficient and natural access to information services from any phone. Recently, spoken dialogue systems for widely used applications such as email, travel information, and customer care have moved from research labs into commercial use. These applications can receive millions of calls a month. This huge amount of spoken dialogue data has led to a need for fully au...
متن کامل